Potentials and Limits of Super-Resolution Algorithms and Signal Reconstruction from Sparse Data

A common distortion in videos is image instability in the form of chaotic global and local displacements of image frames caused by camera instability, fluctuations in the refraction index of the light propagation media and similar factors. Such videos that very frequently present moving objects on a stable background contain tremendous redundancy that potentially can be used for image stabilization and perfecting provided reliable separation of stable background from true moving objects. Recently, it was proposed to use this redundancy for resolution enhancement of video through elastic registration, with subpixel accuracy, of segments of video frames which represent stable scenes. The present work is aimed at studying the potentials and limitations of such a resolution enhancement. The work consists of two parts. The first part investigates, by means of computer simulation, the influence on the degree of the achievable resolution enhancement such imaging parameters as the camera fill factor, of intensity of fluctuations of pixel displacements and of the number of image frames used for fusion. The essential part of the process of resolution enhancement is signal reconstruction from sparse data accumulated from the set of randomly displaced image frames. Therefore the second part of the work addresses the theory of discrete signal reconstruction from sparse data. Using the discrete sampling theorem, it is shown that given a finite number of sparse image samples one can reconstruct a band-limited, in terms of a selected basis, approximation of the signal with minimal mean squared error. Limitation imposed by different bases to the positions of available sparse signal samples to secure optimal restoration are analyzed and it is shown that low pass band-limited in DFT basis functions can be precisely reconstructed from their arbitrary placed sparse samples. These results are then extended to image reconstruction from limited number of projections or from projections with partly lost samples. POTENTIALS AND LIMITS OF SUPER-RESOLUTION FROM MULTIPLE VIDEO FRAMES For investigating the potentials and resolution of the SR algorithm, a computer simulation program was built. The program’s block diagram is presented in Fig. 1. The algorithm for super-resolution from multiple frames is presented in Fig. 2. Fig. 3 illustrates simulation results of super-resolution. Figure 1: Block diagram of the computer model Figure 2: Block diagram of the super-resolution algorithm a) b) c) Figure 3. Super-resolution from multiple low resolution randomly sampled frames: a) – one of low resolution frames; b) image fused, with super-resolution, from 50 frames c) final output image after iterative interpolation of the image (b) with bandwidth twice as that of initial low resolution frames SIGNAL RECONSTRUCTION FROM SPARSE SAMPLES Band limited signal reconstruction from sparse samples can be implemented in different transform domains. Figure 4, a) shows reconstruction of a signal band limited in Haar wavelet domain, Figure 4, b) shows reconstruction of a signal band limited in Daubechies D4 wavelet domain and Figure 5 illustrates image reconstruction with band limitation in the domain of Discrete Fourier Transform. Figure 6 illustrates image recontruction from sparsely sampled projections. 50 100 150 200 250 300 350 400 450 500 -0.5 0 0.5 1 Original Signal and its Samples 50 100 150 200 250 300 350 400 450 500 -0.5 0 0.5 1 Reconstructed signal by direct matrix inversion a) 50 100 150 200 250 300 350 400 450 500 -0.5 0 0.5 1 Original Signal and its S l 50 100 150 200 250 300 350 400 450 500 -0.5 0 0.5 1 Reconstructed signal by direct matrix i i b) Figure 4 (a) Reconstruction, from sparse samples (shown in blue), of a signal (shown in green) bandlimited in Haar domain; (b) Reconstruction, from sparse samples (shown in blue), of a signal (shown in green) band-limited in Daubechies D4 wave let domain